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Summary 


Background  and  Problem 

Job  requirements  for  Navy  enlisted  electronic  technicians  include  extensive 
knowledge  of  electricity  and  electronic  fundamentals.  Training  in  basic  electricity  and 
electronics  (BE/E)  fundamentals  is  the  first  experience  most  trainees  have  in  the  domain. 
This  curriculum  is  taught  to  over  12,000  trainees  in  21  separate  Navy  Class  “A”  Schools. 
Historically  the  BE/E  material  has  proven  difficult  for  trainees  to  learn  and  has  resulted 
in  high  rates  of  setbacks  and  attrition. 

Innovations  in  computer  based  instruction  and  computer  based  games  indicate  that 
these  approaches  may  prove  to  be  beneficial  for  complex  content  such  as  basic  electricity 
and  electronics.  Gaming  and  game  embellishments  such  as  animated  graphics,  video, 
and  high  quality  sound  are  being  introduced  into  educational  software  because  it  is 
assumed  that  such  features  will  help  sustain  motivation  over  the  long  periods  of  time 
needed  to  achieve  competence  in  complex  skills.  Furthermore,  a  number  of  studies  have 
found  that  computer  based  instruction  is  both  efficient  and  effective. 

Objectives 

The  objective  of  this  effort  was  to  evaluate  alternative  computer-based  instructional 
strategies  for  teaching  complex  technical  content.  Specifically,  the  study  compared 
student  performance  and  motivation  in  three  different  computer-based  instructional 
conditions  with  standard  Navy  classroom  instruction  at  the  Aviation  Avionics  “A” 
School,  Naval  Air  Station,  Memphis,  TN. 

Method 

Eighty  trainees  completed  the  instructional  content  which  covered  4  days  of  an 
electronics  curriculum.  Trainees  were  randomly  assigned  to  one  of  four  instructional 
treatments  designed  to  teach  the  same  content:  Computer  based  drill  and  practice 
instruction  (CBDP,  n  -  13),  enhanced  computer  based  instruction  (ECBI,  n  =  23),  the 
game  (GAME,  n  =  20),  and  the  existing  classroom  instruction  (Cl,  n  =  24).  The  trainees 
were  administered  the  schoolhouse  comprehensive  test,  a  specially  designed  cognitive 
skills  test,  and  a  motivation  questionnaire  upon  completion  of  the  instruction.  In 
addition,  time  to  complete  the  instruction  was  recorded  and  analyzed. 

The  CBDP  instruction  included  the  graphics  and  visualizations  found  in  the  instructor 
guide  for  classroom  instruction.  In  this  condition,  the  trainee  could  receive  unlimited 
drill  and  practice  in  vocabulary  as  well  as  problem  solving  for  unknown  circuit  values. 
The  CBDP  curriculum  emphasis  was  on  learning  definitions  and  on  learning  procedures 
for  solving  problems. 

The  ECBI  instruction  illustrated  BE/E  concepts  with  computer  generated  graphics 
and  animated  simulations.  As  with  the  CBDP,  the  trainee  could  receive  unlimited  drill 
and  practice  in  vocabulary  and  in  solving  for  unknown  circuit  values.  However,  the 
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emphasis  was  on  learning  how  circuit  components  interact  to  accomplish  circuit 
functions. 

The  GAME  was  designed  to  resemble  popular  arcade  games.  It  was  a  role  playing 
game  in  which  the  learner  assumed  the  role  of  a  character  in  a  fantasy  world.  The 
GAME’S  scenario  centered  on  a  fictional  Navy  ship  that  is  blasted  from  the  future  to  the 
present.  The  mission  of  the  player  was  to  find  the  resources  needed  to  repair  the  ship’s 
systems  and  return  it  to  the  future.  Accomplishing  this  goal  required  that  the  trainee 
traverse  a  number  of  compartments  by  answering  questions  and  solving  problems.  Each 
compartment  addressed  a  single  concept  or  instructional  objective.  To  leave  a 
compartment,  a  trainee  must  have  completed  an  instructional  exercise  that  was  cast  in  the 
fantasy  of  the  game. 

In  the  Classroom  Instruction  (Cl),  instructors  at  the  Avionics  “A”  School  taught  from 
a  Navy  approved  instructor  guide.  The  instructor  guide  was  part  of  a  highly  structured 
curriculum,  which  specified  the  subject  matter,  the  order  of  presentation,  and  the  training 
aids  to  be  used  for  each  lesson. 

Results 

In  general  the  CBDP  and  ECBI  groups  outperformed  the  Cl  and  GAME  groups  on  all 
measures.  The  findings  support  the  hypotheses  that  (1)  the  CBDP  and  ECBI  computer 
based  instructional  strategies  are  more  effective  than  current  classroom  instruction,  (2) 
the  CBDP  and  ECBI  computer  based  instructional  strategies  are  more  efficient  than 
current  classroom  instruction,  (3)  all  three  computer  based  instructional  strategies  are 
more  motivating  than  current  classroom  instruction,  and  (4)  there  are  differences  in 
effectiveness  and  efficiency  among  the  computer  based  instructional  strategies  with  the 
GAME  condition  performing  no  better  than  the  Cl  condition.  When  there  were 
differences  between  the  ECBI  and  CBDP  groups,  the  ECBI  group  performed  better. 
However,  performance  on  the  quantitative  knowledge  component  of  the  cognitive  skills 
test  was  below  passing  for  all  groups. 

Conclusions 

The  findings  have  implications  for  the  design  and  development  of  computer  based 
instruction.  First,  instructional  developers  should  not  overdevelop  their  curriculum.  The 
complex  visualizations  used  in  the  ECBI  lessons  did  not  significantly  improve 
performance  on  the  concept  definition  and  symbol  identification  tasks.  Unnecessary 
visualizations  serve  only  to  increase  course  length  and  development  costs.  However, 
visualizations  did  prove  effective  in  the  ECBI  group  when  they  were  used  to  illustrate 
cause  and  effect  relationships  and  make  invisible  concepts  visible.  Second,  the  poor 
performance  on  the  quantitative  knowledge  component  of  the  cognitive  skills  test  may  be 
attributable  to  trainees  not  having  enough  opportunities  for  practice.  All  trainees 
received  the  same  fixed  amount  of  practice.  If  they  were  unable  to  complete  three 
problems  correctly,  they  were  “told”  that  they  should  practice  more  problems  of  that  type 
on  their  own  and  advance  to  the  next  topic.  Apparently,  most  trainees  did  not  practice 
additional  problems  on  their  own.  The  computer  based  instructional  programs  could 
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have  required  trainees  to  practice  until  a  mastery  criterion  had  been  achieved.  Finally, 
the  GAME  was  extraordinarily  expensive  to  produce  and  difficult  to  debug.  Because  of 
a  lack  of  experience  in  developing  training  oriented  games,  the  GAME  was 
conceptualized  without  regard  for  potential  programming  challenges.  The  result  was  that 
the  GAME  was  no  more  effective  than  classroom  instruction,  but  was  much  more 
expensive.  Ideally,  game  components  should  provide  the  motivation  needed  for 
instruction  while  at  the  same  time  not  distracting  the  trainees.  On  the  other  hand,  an 
instructional  design  that  maximizes  instruction  often  interrupts  the  flow  of  the  game 
fantasy.  Integrating  instruction  and  gaming  requires  a  delicate  balance  between  gaming 
and  instruction  to  prevent  one  component  from  overwhelming  the  other.  There  is  clearly 
a  need  for  more  research  before  games  can  be  recommended  as  a  source  of  instruction. 
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Introduction 


Problem 

Basic  Electricity  and  Electronics  (BE/E)  fundamentals  constitute  the  first 
instructional  modules  in  the  curriculum  of  21  Navy  Class  “A”  Schools.  Historically,  the 
BE/E  material  has  proven  difficult  for  students  to  learn  and  has  resulted  in  high  setback 
and  attrition  rates.  In  recent  years,  some  “A”  Schools  have  experienced  setback  rates  as 
high  as  60  percent  and  attrition  rates  of  up  to  25  percent  for  the  BE/E  portion  of  the 
training.  Research  at  this  Center  has  found  that  BE/E  graduates  failed  over  half  of  the 
items  on  applied  skills  tests  derived  directly  from  the  BE/E  course  materials  (Parchman, 
Ellis,  &  Montague,  1990).  Recent  innovations  in  computer  based  instruction  and 
computer  based  games  indicate  that  these  approaches  may  prove  to  be  beneficial  for 
complex  content  such  as  basic  electricity  and  electronics.  For  example,  gaming  has 
become  an  increasingly  prevalent  technique  used  by  instructional  courseware  developers 
to  increase  time  on  task  and  motivation  (Malone,  1981;  Dunne,  1984;  Lepper,  1985; 
Lepper  &  Chabay,  1985;  Lepper  &  Malone,  1987;  Malone  &  Lepper,  1987;  Brown, 
1994).  Numerous  reviews  have  found  that  computer  based  instruction  is  both  efficient 
and  effective.  (Bangert-Downs,  Kulik,  &  Kulik,  1985;  Chambers  &  Spreecher,  1980; 
Kulik,  Bangert,  &  Williams,  1983;  Kulik,  Kulik,  &  Bangert-Downs,  1985;  Kulik,  Kulik, 
&  Cohen,  1980;  Kulik  &  Kulik,  1986;  Kulik  &  Kulik,  1987;  Neimiec  &  Walberg,  1987; 
Orlansky  &  String,  1979;  Roblyer,  1988) 

Background 

The  efficacy  of  computer  based  instruction  (CBI)  has  been  evaluated  and  is  well 
documented.  Kulik  (1994)  reviewed  the  data  from  a  large  number  of  studies  and 
concluded  that: 

•  Students  learn  more.  CBI  improved  achievement  scores  by  0.35  standard 
deviations,  or  from  the  50th  percentile  to  the  64th  percentile,  for  all  studies. 

•  Students  learn  in  less  time.  CBI  reduced  instruction  time  by  an  average  of  34 
percent  in  17  studies  of  college  instruction  and  24  percent  in  15  studies  of  adult 
education. 

•  Students  like  their  classes  better.  CBI  improved  attitudes  toward  instruction  by 
0.28  standard  deviations  in  17  studies. 

Recent  research  has  found  that  teaching  content  using  computer  games  can  also  be 
beneficial.  Gaming  and  game  embellishments  such  as  animated  graphics,  video,  and 
high  quality  sound  are  being  introduced  into  educational  software  because  it  is  assumed 
that  such  features  will  help  sustain  motivation  over  the  long  periods  of  time  needed  to 
achieve  competence  in  complex  skills  (Randel,  Morris,  Wetzel,  &  Whitehill,  1992). 
Their  findings  indicate  that  when  games  were  used  in  instruction: 
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•  The  trainees  learned  the  same  amount  as  in  traditional  classroom 
instruction.  In  63  comparisons,  they  found  that  achievement  improved  in  22 
studies,  remained  the  same  in  38  studies,  and  decreased  in  three  studies. 

•  Students  liked  their  classes  better.  In  12  of  14  studies,  trainees  reported  more 
interest  in  the  game  based  instruction  than  in  traditional  classroom  instruction. 

•  Students  remembered  better  than  traditional  classroom  instruction.  In  14 

studies  measuring  retention,  they  found  that  retention  was  better  in  10  and 
remained  the  same  in  four. 

The  present  study  evaluates  three  different  computer  based  strategies  for  basic 
electricity  and  electronics  training.  The  first  strategy  is  computer  based  drill  and  practice 
(CBDP).  The  trainee  is  instructed  using  text  passages  and  graphics  with  all  key 
vocabulary  items  in  bold  font.  As  trainees  progress  through  the  instruction,  they  review 
and  practice  vocabulary  and  problem  solving.  Simulations  are  not  used  and  graphics  are 
simple  one  or  two  colored  annotated  line  drawings.  The  instruction  does  not  make  full 
use  of  the  computer’s  capability  to  illustrate  concepts  with  graphics  and  simulations. 

The  second  strategy  is  enhanced  computer  based  instruction  (ECBI).  The  trainee  is 
instructed  using  attractive  and  compelling  text,  graphics,  animation,  and  simulations  to 
illustrate  the  subject  matter.  It  makes  full  use  of  the  computer  to  visually  illustrate 
concepts  and  to  direct  the  trainee’s  attention  to  the  pertinent  aspects  of  the  subject  matter. 
As  with  the  CBDP  environment,  trainees  are  provided  reviews  and  are  quizzed  as  they 
progress  through  the  instruction  using  embedded  questions  and  a  variety  of  problem 
solving  tasks. 

The  third  strategy  is  a  computer  based  adventure  game  (GAME).  It  implements  the 
elements  found  in  successful  games.  GAME  is  a  role  playing  game  where  the  trainee 
assumes  the  role  of  a  character  in  a  fantasy  world.  The  fantasy  world  is  a  battleship  from 
the  future.  The  battleship  is  composed  of  compartments  and  the  trainee  must  visit  all  of 
the  compartments.  Within  a  compartment,  a  trainee  can  explore  and  discover  objects  and 
various  sorts  of  characters.  To  “escape”  from  a  compartment,  the  trainee  must  complete 
an  exercise  that  is  cast  within  the  fantasy  of  the  game.  The  instruction  for  a  compartment 
consists  of  an  advanced  organizer,  execution  of  the  exercise,  and  a  summary  of  the 
concepts  conveyed  by  the  exercise.  The  advanced  organizer  and  summary  are  typically 
delivered  by  a  fantasy  character  who  serves  as  the  trainee’s  guide  and  advisor  through  the 
adventure. 

Research  Questions 

The  research  questions  are: 

•  Are  the  computer  based  instructional  strategies  more  effective  than  current 
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Are  the  computer  based  instructional  strategies  more  efficient  than  current 
classroom  instruction? 


•  Are  the  computer  based  instructional  strategies  more  motivating  than  current 
classroom  instruction? 

•  Are  there  differences  in  effectiveness  and  efficiency  among  the  computer  based 
instructional  strategies? 

Objective 

The  objective  is  to  evaluate  alternative  instructional  strategies  for  teaching  complex 
technical  content.  Specifically,  the  study  compares  student  performance  and  motivation 
in  four  different  instructional  conditions:  (1)  computer  based  drill  and  practice,  (2) 
enhanced  computer  based  instruction,  (3)  computer  based  instructional  games,  and  (4) 
traditional  classroom  instruction. 


Method 


Subjects 

The  subjects  were  88  trainees  at  the  Aviation  Avionics  “A”  School,  Naval  Air  Station 
at  Memphis,  Tennessee.  Trainees  were  selected  to  become  avionics  technicians  on  the 
basis  of  their  Armed  Services  Vocational  Aptitude  Battery  (ASVAB)  scores;  therefore, 
the  subject  population  was  relatively  homogeneous.  The  trainees  were  assigned 
randomly  by  class  to  one  of  the  four  instructional  conditions:  CBDP  ( n  =  14),  ECBI  (n  = 
25),  Game  ( n  =  25),  Classroom  (n  =  24).  All  trainees  had  completed  the  first  5  weeks  of 
the  Avionics  “A”  School  curriculum  and  had  taken  five  end-of-week  tests.  Trainees  in 
the  CBDP,  ECBI,  and  GAME  conditions  then  received  experimental  instruction  outside 
of  the  standard  “A”  school  classroom.  Upon  completion  of  the  experimental  curriculum, 
they  returned  to  the  regular  course. 

Trainees  with  incomplete  or  missing  data  were  omitted  from  the  study.  Trainees 
were  omitted  because  of  sickness,  legal  problems,  or  were  dropped  from  the  course  for 
achievement,  administrative,  or  disciplinary  reasons.  Trainees  with  complete  data 
numbered  80.  The  final  number  per  instructional  condition  was  CBDP  (n  =  13),  ECBI  (n 
=  23),  Game  ( n  -  20),  Classroom  (n  -  24). 

Design 

The  research  design  selected  was  a  quasi-experimental,  four  group,  posttest-only 
design.  The  four  treatment  groups  were: 

1.  Computer  Based  Drill  and  Practice:  Trainees  received  the  CBI  instruction  and 
were  then  given  a  battery  of  dependent  measures. 
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2.  Enhanced  Computer  Based  Instruction:  Trainees  received  the  ECBI 
instruction  and  were  then  given  a  battery  of  dependent  measures. 

3.  GAME:  Trainees  received  the  GAME  instruction  and  were  then  given  a  battery 
of  dependent  measures. 

4.  Classroom  Instruction:  Trainees  received  the  regular  course  of  instruction  and 
were  then  given  a  battery  of  dependent  measures. 

Dependent  Measures 

Pre-Measures 

A  pre-training  measure  of  performance  was  calculated  for  each  subject.  This 
measure  was  the  mean  score  achieved  on  the  five  weekly  Aviation  Avionics  “A”  School 
course  tests  completed  prior  to  beginning  the  experimental  curriculum. 

Post  Measures 

The  trainees  were  administered  two  different  achievement  tests  upon  completion  of 
the  instruction:  a  schoolhouse  developed  test  (Schoolhouse  Test)  and  a  test  developed  by 
NPRDC  (NPRDC  Test). 

The  Schoolhouse  Test  tested  13  of  the  25  task  statements  covered  by  the  course 
objectives.  The  NPRDC  Test  tested  all  25  task  statements.  The  NPRDC  Test  had  four 
parts: 

1.  Knowledge  of  definitions. 

2.  Knowledge  of  symbols. 

3.  Qualitative  knowledge. 

4.  Quantitative  knowledge. 

The  trainees  also  were  administered  the  ARCS  motivation  questionnaire  developed 
by  Keller  (1989).  The  ARCS  was  used  to  measure  the  trainees’  perceptions  of  the 
motivational  characteristics  of  the  instruction.  It  has  four  subscales  designed  to  measure 
the  degree  to  which  the  instructional  materials  address  the  motivational  components  of 
attention,  relevance,  confidence,  and  satisfaction.  The  reliability  of  the  ARCS  given  by 
Cronbach’s  alpha  is  .89  for  attention,  .90  for  confidence,  .81  for  relevance,  .92  for 
satisfaction,  and  .96  for  overall  motivation. 

Instructional  Content 

The  content  covered  4  days  of  the  143  day  Aviation  Avionics  “A”  School  electronics 
curriculum.  The  4  days  of  instruction  included  lessons  on  capacitance,  capacitors,  and 
resistance-capacitance  (RC)  circuits  using  direct  current  (DC)  as  the  source.  The 
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instruction  covered  two  learning  objectives  and  25  specific  learning  tasks.  The  tasks 
included  knowledge  of  terms  and  symbols,  physical  characteristics  of  electrical 
components,  and  safety  precautions,  as  well  as  problem  solving. 

Instructional  Treatments 

There  were  four  instructional  treatments.  All  four  treatments  were  designed  to  teach 
the  same  content.  The  computer  based  drill  and  practice  instruction  (CBDP)  and 
enhanced  computer  based  instruction  (ECBI)  treatments  were  developed  by  the  authors  at 
the  Navy  Personnel  Research  and  Development  Center  (NPRDC).  The  game  (GAME) 
treatment  was  developed  by  NPRDC  and  contractor  personnel.  The  classroom 
instruction  (Cl)  was  the  existing  lecture-based  class  taught  by  the  school  instructors. 

Computer  Based  Drill  and  Practice  Instruction 

The  CBDP  instruction  was  designed  to  parallel  the  current  classroom  instruction.  It 
included  only  the  graphics  and  visualizations  found  in  the  instructor  guide  for  the 
capacitance  and  RC  portions  of  current  instruction.  In  addition,  the  trainee  could  receive 
unlimited  drill  and  practice  in  vocabulary  and  problem  solving  for  unknown  circuit 
values.  The  emphasis  of  the  CBDP  curriculum  was  on  learning  definitions,  concepts, 
and  procedures  for  solving  problems. 

The  CBDP  curriculum  was  a  series  of  lessons  consisting  of  a  text  passage  and 
questions  on  a  particular  concept  or  procedure.  When  the  trainee  answered  correctly,  the 
system  displayed  “Correct.”  The  feedback  after  an  incorrect  answer  depended  on  the 
number  of  tries.  If  the  trainee’s  first  response  was  incorrect,  the  system  displayed 
“Wrong-Try  Again.”  If  the  second  attempt  was  incorrect,  the  system  displayed  the 
message  “Wrong”  and  showed  the  trainee  an  annotated  step  by  step  solution  to  the 
problem.  “HELP”  was  available  at  all  times.  If  the  trainee  asked  for  “HELP,”  the  system 
displayed  an  annotated  problem  that  was  similar  to  the  problem  being  studied.  Each 
trainee  was  required  to  repeat  each  problem  type  a  minimum  of  three  times  before  being 
advanced.  However,  trainees  could  opt  to  work  as  many  additional  problems  as  they 
thought  necessary. 

The  CBPD  incorporated  four  drill  and  practice  learning  strategies.  The  first  strategy, 
the  vocabulary  definition  strategy,  simulated  flash  cards.  This  strategy  presented  each 
vocabulary  word  in  the  lesson  in  succession  with  its  full  definition.  This  allowed  the 
trainee  to  read  through  each  definition  before  being  quizzed.  The  second  strategy  was 
definition  building.  For  this  strategy,  the  trainee  constructed  the  definition  of  a 
vocabulary  item  by  selecting  phrases  from  a  menu.  The  menu  contained  one  phrase  from 
a  target  definition  and  phrases  from  other  related  words  in  the  lesson.  The  third  strategy 
was  hidden  multiple  choice.  For  this  strategy,  the  trainee  viewed  each  definition  one  at  a 
time  and  indicated  whether  or  not  it  was  the  correct  definition.  The  fourth  strategy  was 
visible  multiple  choice.  In  this  strategy,  the  trainee  selected  the  correct  definition  from  a 
list  of  related  word  items  found  in  the  lesson. 
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The  CBDP  also  provided  drill  and  practice  in  both  qualitative  and  quantitative 
problem  solving.  For  qualitative  exercises,  the  trainee  read  a  description  on  how  a  circuit 
worked  and  was  asked  questions  about  qualitative  relationships.  For  quantitative 
exercises,  the  trainee  was  asked  to  apply  the  appropriate  formula  to  solve  for  unknown 
circuit  values.  A  class  of  14  trainees  was  assigned  to  the  CBDP  treatment.  The  trainees 
received  CBDP  until  they  had  completed  the  instruction  on  capacitors,  capacitance,  and 
RC  circuits. 

Enhanced  Computer  Based  Instruction 

The  Enhanced  Computer  Based  Instruction  (ECBI)  took  advantage  of  the  computer 
to  illustrate  concepts  with  graphics  and  animated  simulations.  As  with  the  CBDP,  the 
trainee  could  receive  unlimited  drill  and  practice  in  vocabulary  and  solving  for  unknown 
circuit  values.  However,  the  emphasis  was  on  learning  how  circuit  components  interact 
to  accomplish  circuit  functions. 

The  ECBI  treatment  was  similar  to  the  CBDP  treatment.  The  curriculum  was  a  series 
of  lessons  consisting  of  a  text  passage,  graphics,  animated  simulations,  and  questions. 
When  questions  were  asked  and  the  trainee  gave  a  correct  answer,  the  system  displayed 
“Correct.”  The  feedback  after  an  incorrect  answer  depended  on  the  number  of  tries.  If 
the  trainee’s  first  response  was  incorrect,  the  system  displayed  “Wrong-Try  Again.” 
After  three  attempts,  the  system  told  the  trainee  to  review  the  explanatory  text.  “HELP” 
was  available  at  all  times.  If  the  trainee  requested  help,  the  system  displayed  an 
annotated  problem  that  was  similar  to  the  problem  being  studied.  Trainees  were  required 
to  repeat  each  problem  type  at  least  three  times.  If  a  trainee  failed  three  out  of  five 
problems,  the  program  instructed  the  trainee  to  review  the  explanatory  text  and  examples 
and  work  more  problems.  As  with  the  CBDP,  trainees  could  work  as  many  problems  as 
they  thought  necessary.  A  class  of  25  trainees  were  assigned  to  the  ECBI  treatment.  The 
trainees  received  ECBI  until  they  had  completed  the  instruction  on  capacitors, 
capacitance,  and  RC  circuits. 

GAME 

The  GAME  was  designed  to  resemble  popular  games  such  as  “Kings  Quest  V” 
(Williams,  1990)  and  “3  in  Three”  (Johnson,  1990).  Like  these,  the  GAME  was  a  role 
playing  game  in  which  the  learner  assumed  the  role  of  a  character  in  a  fantasy  world. 
The  GAME’S  scenario  centered  on  a  Navy  ship  that  was  blasted  from  the  future  to  the 
present.  The  mission  of  the  player  was  to  find  the  resources  needed  to  repair  the  ship’s 
systems  and  return  to  the  future.  Accomplishing  this  goal  required  that  the  trainee 
traverse  a  number  of  compartments.  Each  compartment  addressed  a  single  concept  or 
instructional  objective.  To  leave  a  compartment,  a  trainee  must  have  completed  an 
instructional  exercise  that  was  cast  in  the  fantasy  of  the  game.  The  trainee’s  path  through 
the  compartments  was  constrained  by  the  structure  of  the  content  to  be  learned. 

A  typical  instructional  sequence  for  a  single  compartment  consisted  of  an  advanced 
organizer,  exercises,  and  a  summary  of  the  topics  to  be  covered  in  the  compartment. 
Depending  on  the  task,  some  or  all  of  the  trainees  would  require  more  than  the  brief 
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introduction  in  the  advanced  organizer  to  complete  the  exercise.  In  these  cases, 
additional  instruction  and  information  was  provided  through  a  “classroom”  that  was 
available  to  the  trainee  at  any  point  in  the  game.  The  “classroom”  operated  in  two 
modes;  exploratory  mode  and  direct  mode.  In  exploratory  mode,  the  trainee  was  free  to 
explore,  in  a  hypertext  environment,  the  topics  addressed  in  the  compartment.  The 
hypertext  environment  consisted  of  a  series  of  topics  and  a  presentation  for  each  topic. 
Within  each  presentation  were  links  to  other  topics  and  presentations.  In  direct  mode,  the 
trainee  received  linear  CBI  on  the  topics  addressed  in  the  compartment.  A  linear  CBI 
lesson  consisted  of  frames  containing  instruction  on  the  compartment’s  focus.  Embedded 
questions  were  used  to  maintain  attention  and  ensure  comprehension.  Both  modes  took 
advantage  of  the  computer  to  illustrate  concepts  with  graphics  and  simulations  such  as 
those  employed  by  Hollan,  Hutchins,  and  Weitzman  (1984).  Schematic  diagrams  were 
used  to  illustrate  circuits.  Practice  exercises  in  problem  solving  and  reasoning  required 
the  trainee  to  solve  problems  and  conduct  observations  on  simulated  circuits.  A  class  of 
25  trainees  were  assigned  to  the  GAME  treatment.  The  trainees  played  until  they  had 
completed  the  instruction  on  capacitors,  capacitance,  and  RC  circuits. 

Classroom  Instruction 

In  the  Classroom  Instruction  (Cl)  condition,  instructors  at  the  Avionics  “A”  School 
taught  from  a  Navy  approved  instructor  guide.  The  instructor  guide  provided  a 
structured  curriculum,  which  specified  the  subject  matter,  the  order  of  presentation,  and 
the  training  aids  to  be  used  for  each  lesson.  Instructors  were  encouraged  to  personalize 
each  lesson.  Personalization  consisted  primarily  of  providing  examples  from  the 
instructors  personal  experiences  that  supported  the  curriculum.  The  instructor  was 
expected  to  be  a  subject  matter  expert.  Typically,  questions  raised  in  the  classroom  were 
resolved  by  the  instructor.  The  trainees  were  provided  with  trainee  guides  to  assist  in 
learning.  Trainees  who  had  difficulties  were  counseled  and  given  additional  instruction 
and/or  time  to  study.  A  class  of  24  trainees  was  assigned  to  the  Cl  treatment.  The 
trainees  were  instructed  by  Navy  instructors  until  they  had  completed  the  instruction  on 
capacitors,  capacitance,  and  RC  circuits. 

Procedure 

Trainees  were  under  instruction  from  0700  to  1530  daily  with  a  1  hour  break  for 
lunch.  Since  the  CBDP,  ECBI,  and  the  GAME  groups  were  self-paced,  trainees  were 
told  they  could  break  whenever  they  felt  it  was  necessary.  All  trainees  in  a  particular 
class  were  assigned  the  same  instructional  program.  All  trainees  were  told  that  they 
would  be  learning  about  capacitors,  capacitance,  and  RC  circuits  using  computer  based 
instruction.  It  was  emphasized  that  their  performance  counted  toward  their 
success/failure  in  the  course.  The  trainees  understood  that  their  grade  would  be  entered 
into  their  military  record.  Trainees  who  failed  the  Navy  developed  comprehensive 
progress  test  were  either  “set-back”  to  repeat  the  Cl  curriculum  or  remediated  with 
tutorial  instruction  and  then  retook  the  exam.  The  trainees  were  told  that  the  instructors 
would  not  be  available  for  supplementary  tutoring  and  that  the  trainees  were  responsible 
for  completing  all  of  the  required  homework  found  in  the  study  guides.  The  resources 
available  were  two  classroom  instructors,  study  guides,  fellow  classmates,  referenced 
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publications,  paper,  pencils,  and  calculators  in  accordance  with  Navy  course  regulations. 
The  trainee  groups  were  not  aware  of  any  differences  in  the  experimental  programs,  and 
they  were  not  aware  that  performance  was  being  monitored.  When  subjects  completed 
the  program,  they  were  allowed  to  review  the  material  before  being  tested. 

Results 


Pre-Measures 

An  analysis  of  variance  (ANOVA)  compared  the  four  groups  on  the  mean  score 
achieved  on  all  “A”  school  course  comprehensive  tests  (CCT)  prior  to  beginning  the 
experimental  curriculum.  There  were  no  significant  differences  among  the  groups  on 
this  measure  ( p  <  .05).  This  measure  was  used  as  a  covariate  in  subsequent  analyses. 

Post-Measures 

Schoolhouse  Test 

An  analysis  of  covariance  was  performed  on  the  total  percent  correct  for  the 
Schoolhouse  Test  with  the  CCT  as  a  covariate.  Table  1  presents  the  adjusted  means  and 
standard  deviations  of  the  Schoolhouse  Test.  The  analysis  showed  no  differences  among 
the  treatment  groups  (p  <  .05).  One  reason  is  that  only  half  of  the  tasks  covered  by  the 
course  objectives  was  assessed  in  the  Schoolhouse  Test.  Further,  an  analysis  of  the  test 
showed  that  many  of  the  more  complicated  tasks  and  concepts  were  not  tested. 

Table  1 

Mean  Percent  Correct  Scores  and  Standard  Deviations  for  the 
Schoolhouse  Test  and  NPRDC  Sub-Tests 


CBDP 

ECBI 

GAME 

Cl 

Schoolhouse  Test 

84.17 

SD  =  12.40 

82.00 

SD  =  10.87 

84.78 

SD  =  10.70 

88.29 

SD  =  10.21 

Knowledge  of  Definitions 

83.33 

SD  =  12.55 

80.00 

SD  =  14.67 

62.00 

SD  =  14.55 

58.00 

SD  =  25.00 

Knowledge  of  Symbols 

92.00 

SD  =  10.81 

84.54 

SD  =  15.00 

83.27 

SD  =  14.45 

77.27 

SD  =  25.98 

Qualitative  Knowledge 

78.71 

SD  =  13.19 

81.65 

SD  =  13.14 

66.20 

SD  =  18.43 

70.92 

SD  =  16.04 

Quantitative  Knowledge 

54.61 

SD  =  29.00 

57.92 

SD  =  23.61 

52.06 

SD  =  22.15 

60.94 

SD  =  24.27 
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NPRDC  Test 


A  multivariate  analysis  of  variance  (MANOVA)  with  the  CCT  as  a  covariate  was 
performed  on  the  percent  correct  scores  for  each  of  the  four  sub-tests  of  the  NPRDC  test. 
There  was  a  significant  effect  indicating  significant  differences  among  groups  ( p  <  .01). 

Univariate  ANOVAs  were  then  calculated  for  each  of  the  four  sub- tests  of  the 
NPRDC  test  to  determine  group  differences.  The  analysis  of  the  knowledge  of 
definitions  sub-test  showed  significant  differences  among  the  groups  (p  <  .01).  Post  hoc 
comparisons  revealed  that  trainees  in  CBDP  and  ECBI  conditions  did  not  differ. 
However,  CBDP  and  ECBI  trainees  significantly  outperformed  trainees  in  Cl  and  GAME 
conditions  (p  <  .01),  which  did  not  differ.  The  analysis  performed  on  the  symbol’s  sub¬ 
test  showed  no  significant  differences  among  the  instructional  conditions.  The  analysis 
for  the  qualitative  knowledge  sub-test  found  significant  differences  in  performance 
among  the  groups  (p  <  .01).  Post  hoc  comparisons  revealed  that  CBDP  and  ECBI 
trainees  performed  significantly  better  than  trainees  in  the  GAME  condition  {p  <  .01). 
Further  ECBI  trainees  scored  significantly  better  than  trainees  in  the  Cl  condition  {p  < 
.01).  There  were  no  differences  between  the  Cl  and  GAME  conditions  or  the  CBDP  and 
ECBI  conditions.  Finally,  the  analysis  for  the  quantitative  reasoning  sub-test  showed  no 
significant  differences  among  the  conditions.  Table  1  presents  the  means  and  standard 
deviations  for  these  analyses. 

ARCS  Motivation  Questionnaire 

A  multivariate  analysis  of  variance  (MANOVA)  was  performed  on  the  scores  for 
each  of  the  four  scales  of  the  ARCS  motivation  questionnaire  (Attention,  Confidence, 
Relevance,  and  Satisfaction).  There  was  a  significant  effect  indicating  significant 
differences  among  groups  (p  <  .01).  Table  2  presents  the  means  for  the  motivation 
analysis. 


Table  2 

Mean  Scores  for  ARCS  Motivation  Questionnaire  Scales 


CBDP 

ECBI 

GAME 

Cl 

Attention 

3.09 

3.71 

3.46 

3.29 

Relevance 

3.44 

3.55 

3.36 

3.34 

Confidence 

3.04 

3.59 

2.65 

3.11 

Satisfaction 

2.81 

3.23 

2.93 

3.04 

Univariate  ANOVAs  were  calculated  for  each  of  the  four  scales.  For  the  Attention 
scale,  there  were  significant  differences  among  groups  (p  <  .01).  Post  hoc  comparisons 
showed  that  the  ECBI  and  GAME  conditions  captured  significantly  more  of  the  trainees 
attention  than  CBDP  and  Cl  conditions  (p  <  .01).  There  were  no  significant  differences 
between  CBDP  and  Cl  groups  or  between  the  ECBI  and  GAME  groups.  There  were  no 
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differences  among  the  groups  on  the  Relevance  scale.  The  analysis  for  the  Confidence 
scale  showed  significant  differences  between  groups  (jp  <  .01).  Post  hoc  comparisons 
revealed  ECBI  trainees  were  significantly  more  confident  than  trainees  in  the  other  three 
conditions  (p  <  .03).  There  were  no  differences  among  the  Cl,  GAME,  and  CBDP 
groups.  There  were  no  differences  among  the  groups  on  the  Satisfaction  scale. 

Time 

Table  3  presents  the  means  and  standard  deviations  for  the  instructional  time  for  the 
CBDP,  ECBI,  and  Cl  conditions.  Unfortunately,  there  was  an  error  in  the  computer 
program,  and  as  a  result,  time  data  for  the  GAME  group  was  lost.  The  data  show  that 
trainees  in  the  CBDP  and  ECBI  groups  spent  26  percent  and  28  percent  less  time  in 
instruction,  respectively,  than  trainees  in  Cl  group.  Field  notes  taken  by  investigators 
during  the  study  indicate  that  trainees  in  the  GAME  group  spent  about  the  same  amount 
of  time  completing  the  instruction  as  trainees  in  Cl  group. 

Table  3 

Mean  Time  in  Hours  and  Standard  Deviations  for  Instructional 
Time  for  the  CBDP,  ECBI,  and  Cl  Groups 


CBDP 

ECBI 

CI 

16.37  Hours 

SD  =  2.66 

15.93  Hours 

SD  =  3.24 

22.00  Hours 

SD  =  0.00 

Discussion 

The  primary  findings  are  that  the  CBDP  and  ECBI  groups  in  general  out  performed 
the  Cl  and  GAME  groups  on  all  measures:  course  tests,  motivation,  and  time.  “Yes”  is 
the  answer  to  all  four  research  questions. 

•  Are  the  computer  based  instructional  strategies  more  effective  than  current 
classroom  instruction? 

•  Are  the  computer  based  instructional  strategies  more  efficient  than  current 
classroom  instruction? 

•  Are  the  computer  based  instructional  strategies  more  motivating  than  current 
classroom  instruction? 

•  Are  there  differences  in  effectiveness  and  efficiency  among  the  computer  based 
instructional  strategies? 

When  there  were  differences  between  the  CBDP  and  ECBI  groups,  the  ECBI  group 
performed  better.  However,  it  should  be  noted  that  performance  on  the  quantitative  sub- 
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test  of  the  NPRDC  test  was  low  for  all  groups,  and  no  group  achieved  a  mean  passing 
score. 

The  differences  between  the  self-paced  computer  groups  (CBDP  and  ECBI)  and  the 
Cl  group  were  expected  and  are  consistent  with  a  large  number  of  previous  studies  (e.g., 
Kulik,  Kulik,  &  Shwalb,  1986).  However,  the  reasons  why  the  ECBI  condition  generally 
performed  better  and  why  the  GAME  condition  failed  to  do  better  than  the  much  more 
cost  effective  Cl  condition  need  to  be  examined. 

ECBI  Group  Performance 

ECBI  trainees  generally  learned  more  effectively,  more  efficiently,  and  were  more 
motivated  than  the  other  three  groups.  There  are  several  possible  explanations  for  these 
results.  First,  the  ECBI  treatment  was  more  task  oriented  while  the  CBI  and  Cl  were 
more  topic  oriented.  Task  oriented  instruction  has  been  found  to  be  more  effective  in 
learning,  retention,  and  performance  than  topic  oriented  instruction  (Collins,  Brown,  & 
Newman,  1989;  Cognition  and  Technology  Group  at  Vanderbilt,  1990;  Semb  &  Ellis, 
1994).  Second,  the  ECBI  incorporated  visualization  techniques  and  simple  simulations 
to  illustrate  cause  and  effect  relationships  and  make  invisible  concepts  visible.  Research 
on  learning  from  text  and  on  computer  based  learning  environments  shows  that 
simulation  and  visualization  aid  learning  and  retention  if  they  supplement  the  curriculum 
in  meaningful  ways.  Third,  the  ECBI  provided  the  trainees  with  elaborations  to  illustrate 
how  concepts  and  events  are  structured  and  function.  Research  on  learning  has  shown 
that  elaborations  enhance  a  trainee’s  conceptual  model  of  the  curriculum.  All  of  these 
factors  are  likely  to  have  contributed  to  the  enhanced  performance  and  motivation  of 
ECBI  trainees. 

GAME  Group  Performance 

Games  and  simulations  have  a  long  tradition  of  use  in  the  military  where  a  simulated 
environment  is  substituted  for  the  “real  thing  ”  when  practice  with  the  “real  thing”  is  too 
expensive  or  dangerous.  The  GAME  condition  in  this  study  departed  from  this  tradition 
in  that  the  game  itself  was  the  primary  source  of  instruction.  That  is,  the  game  presented 
all  the  facts,  concepts,  principles,  and  generalizations  needed  for  instruction  in  the 
domain  of  capacitors,  capacitance,  and  RC  circuits.  This  approach  resulted  in 
performance  that  was  no  better  than  standard  Navy  classroom  instruction.  There  are 
several  possible  explanations  for  this  finding.  The  design  goal  for  the  GAME  was  to 
make  it  “competitive”  with  the  best  in  the  commercial  market.  This  may  have  been  a 
poor  design  choice.  Additionally,  the  GAME  generally  suffered  from  technical  flaws 
including  software  bugs,  inaccurate  technical  content,  poor  instructional  techniques,  and 
interface  problems.  Many  artistic  effects  were  included  that  contributed  little  to  its 
effectiveness.  For  example,  in  the  GAME,  each  trainee  was  able  to  select  a  caricature  of 
him  or  herself.  The  caricature  “walked”  through  each  compartment  and  picked  up  items, 
opened  doors,  etc.  This  caricature  served  no  instructional  purpose  and  was  only  a 
gaming  embellishment.  However,  it  was  costly  to  produce,  difficult  to  program,  and  was 
the  source  of  many  program  bugs.  In  retrospect,  the  caricature  was  probably 
unnecessary.  As  suggested  by  Halff  (1994),  the  GAME  should  not  have  been  designed 
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to  compete  with  the  commercial  market,  but  with  the  instructor,  the  transparency,  and  the 
chalkboard.  There  is  a  need  for  research  that  better  delimits  the  characteristics  of  a 
“good”  instructional  game.  Given  these  problems,  it  may  be  a  positive  finding  that  the 
GAME  did  no  worse  than  the  Cl. 


Conclusions 

The  findings  have  implications  for  the  design  and  development  of  computer  based 
instruction.  First,  the  CBDP  and  ECBI  groups  were  equally  effective  when  used  to  teach 
symbols  and  definitions.  This  suggests  that  instructional  developers  should  not 
overdevelop  their  curriculum.  The  complex  visualizations  used  in  the  ECBI  lessons  did 
not  significantly  improve  performance  on  these  tasks.  Unnecessary  visualizations  serve 
only  to  increase  course  length  and  development  costs.  However,  visualizations  did  prove 
effective  in  the  ECBI  group  when  they  were  used  to  illustrate  cause  and  effect 
relationships  and  to  make  invisible  concepts  visible.  This  is  supported  by  the  qualitative 
data  , which  indicates  that  the  ECBI  strategy  of  using  visualizations  to  illustrate  cause  and 
effect  relationships  was  more  effective  than  CBDP  and  Cl,  which  used  a  math  model  and 
the  manipulation  of  symbols  as  the  basis  for  instruction. 

Second,  only  a  few  trainees  mastered  the  quantitative  section  of  the  test.  A  review  of 
performance  on  the  practice  problems  indicated  that  trainees  did  not  have  enough 
opportunities  for  practice.  All  trainees  received  the  same  amount  of  practice.  They 
practiced  each  problem  type  a  maximum  of  five  times.  Once  they  completed  three 
problems  of  a  problem  type  correctly,  they  were  advanced  to  the  next  topic.  If  they  were 
unable  to  complete  three  problems  correctly,  they  were  “told”  that  they  should  practice 
more  problems  of  that  type  on  their  own,  but  still  were  advanced  to  the  next  topic. 
Apparently,  most  trainees  did  not  practice  additional  problems  on  their  own  or,  if  they 
did,  they  continued  to  practice  incorrectly.  The  computer  based  instructional  programs 
could  have  required  trainees  to  practice  until  a  mastery  criterion  had  been  achieved. 

Finally,  the  GAME  was  extraordinarily  expensive  to  produce  and  difficult  to  debug. 
The  cost  of  developing  commercial  video/arcade  games  ranges  from  $250,000  to 
$500,000  per  minute.  Because  of  a  lack  of  experience  in  developing  training  oriented 
games,  the  GAME  was  conceptualized  without  regard  for  potential  programming 
challenges.  As  a  result,  the  programming  team  had  difficulty  anticipating  which  aspects 
of  the  design  would  be  difficult  to  program  (Halff,  1994).  There  is  a  need  for  more 
research  before  games  are  used  as  a  source  of  instruction.  Ideally,  game  components 
should  provide  the  motivation  needed  for  instruction  yet  not  distract  the  trainee.  On  the 
other  hand,  an  instructional  design  that  maximizes  instruction  often  interrupts  the  flow  of 
the  game  fantasy.  Integrating  instruction  and  gaming  requires  maintaining  a  delicate 
balance  of  gaming  and  instruction  to  prevent  one  component  from  overwhelming  the 
other. 
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